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HIGHLIGHTS: 

• Intervention : A charter middle school in New York City, serving mainly low-income, minority students. 

• Evaluation Methods : A well-conducted randomized controlled trial, based on the lottery used to 
determine which students are offered admission. 

• Key Findings : By the end of eighth grade - three years after random assignment - increased students’ 
math achievement by 0.53 standard deviations (about P /2 grade levels) and English language arts 
achievement by 0.20 standard deviations (about % of a grade level). 

• Other : A limitation of this study is that it was conducted in a single site - one school in New York City. 
Replication of these findings in a second trial, conducted in another setting, would be desirable to confirm 
that the intervention is effective across various settings where it might normally be implemented. 


I. The Top Tier initiative’s Expert Panel has identified this intervention as Near Top Tier. 

The Panel finds that this intervention meets the “Near Top Tier” evidence standard, defined as: 

Interventions shown to meet almost all elements of the Top Tier standard (i.e., well-conducted 
randomized controlled trials... showing sizable, sustained effects), and which only need one 
additional step to qualify. This category includes, for example, interventions that meet all elements of 
the standard in a single site, and just need a replication trial to confirm the initial findings and 
establish that they generalize to other sites. 

II. Description of the Intervention 

The Promise Academy is a charter middle school, serving predominantly low-income, minority 
students from grades six through eight. The school opened in 2004 as one of the Harlem Children’s 
Zone programs to improve communities and schools in a 97-block area of Harlem, in New York City. 
The school provides an extended school day and year, with coordinated after-school tutoring and 
additional Saturday classes for children struggling in math or English language arts. As a result, 
students spend 50-100% more time in school per year than students in traditional public schools in 
New York City, depending on how far behind they are academically. The school emphasizes 
recruiting and retaining high quality teachers, who are incentivized and evaluated based on their 
success in raising students’ test scores. Students are consistently reminded of the importance of hard 
work in achieving success, and are given rewards for achievement, such as money or trips. The 
school also provides them with free medical, dental, and mental health services; and provides their 
parents with meals, bus fare, and other benefits. 

The school spent $19,272 per pupil during the 2008-2009 school year, compared to an average of 
$16,144 per pupil in New York City middle schools during that year. 1 

Click here to go to the school’s website . 


1 Both figures are direct service expenditures per pupil, which is a comprehensive measure of expenditures for 
services provided directly to students during the school year. The source for the Promise Academy estimate is 
Dobbie et al. 2010. The source for the city average is the New York City Department of Education’s School Based 
Expenditure Reports . 
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III. Evidence of Effectiveness 


This summary of the evidence is based on a systematic search of the literature, and correspondence 
with leading researchers, to identify all well-conducted randomized controlled trials of the Promise 
Academy charter middle school. Our search identified one such trial, as follows. 

Overview of Study Design : Randomized controlled trial, based on the 2005 and 2006 lotteries 
used to determine which students were offered admission to the school. 

612 students were randomly assigned via lottery to (i) a group of “lottery winners” offered 
admission to the Promise Academy in 6 th grade; or (ii) a control group of “lottery losers” not 
offered admission. 

83% of sample members were African American, 15% were Hispanic, and 80% were eligible for 
free or reduced-price lunch. Prior to the study (i.e., 5 th grade), they scored about a half-year 
behind the average New York City student in both math and English language arts achievement. 2 
69% of the lottery winners accepted the offer of admission and enrolled in the Promise Academy. 

Effects three years after random assignment — i.e., end of 8 th grade : 

These are the Academy’s effects on all outcomes that the study measured, and (unless otherwise 
noted) on all lottery winners, including those who enrolled in the Promise Academy and those who 
did not. The effects shown are statistically significant at the 0.05 level unless stated otherwise. 

■ In math : 

> Lottery winners were 27% more likely than lottery losers to be on grade level, as 
measured by the New York state test. 

> Lottery winners scored higher than lottery losers by 0.53 standard deviations, which 
equates to about IV 2 grade levels. 3 

> The effect grew over the three years of the study (from 0.24 standard deviations in year 1 , 
to 0.26 in year 2, to 0.53 in year 3), and was statistically significant in all three years. 

■ In English language arts : 

> Lottery winners were 1 5% more likely than lottery losers to be on grade level, as 
measured by the New York state test. 

> Lottery winners scored higher than lottery losers by 0.20 standard deviations, which 
equates to about % of a grade level. 3 


2 Specifically, they scored 0.25-0.30 standard deviations below the average New York City student in math and 
English language arts, which equates to approximately a half-year of learning in fifth grade for the typical U.S. 
student (see Bloom, Hill, Black, and Lipsey 2008, referenced at the end of this summary). 

3 Specifically, the average annual achievement gain for U.S. students between the end of seventh grade and the end 
of eighth grade on seven nationally normed tests is 0.32 standard deviations in math and 0.26 in reading (see Bloom, 
Hill, Black, and Lipsey 2008, referenced at the end of this summary). The difference in achievement between 
lottery winners and losers, shown above, is about 1.6 times this amount in math, and 0.75 times this amount in 
reading. 
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> The effect grew over the three years of the study (from 0.02 standard deviations in year 1, 
to 0.03 in year 2, to 0.20 in year 3). It was statistically significant only in year 3. 

■ Absences : Lottery winners were absent 3.9 fewer days than lottery losers during the first six 
months of 8 th grade (the time period when absence data are collected by the city). 

■ Percent entering 8 th grade “on-time” for their age : There was no difference between lottery 
winners and losers on this measure. 

The effects on the subsample of lottery winners who actually accepted the admission offer and 

enrolled in the Academy were 45% larger than the effects shown above. 4 

Discussion of Study Quality : 

■ The study had a reasonably long-term follow-up: three years after random assignment. 

■ The lottery winners and lottery losers in the three-year follow-up sample were similar in their 
observable pre -program characteristics (e.g., demographics, prior academic achievement). 

■ The study evaluated the Promise Academy as it normally operates in Harlem Children’s Zone 
in New York City, thus providing evidence of its effectiveness under real-world 
implementation conditions. We note, however, that the Academy only opened in 2004; thus 
the effects shown above apply to its early years of operation. 

■ The study appropriately measured outcomes for all students who won the lottery, regardless 
of whether or how long they actually attended the Promise Academy (i.e., the study used an 
“intention-to-treat” analysis). 

■ The study measured all outcomes using administrative data from the New York City 
Department of Education, including (for achievement outcomes) student scores on the state 
tests. 

■ Study limitations include the following : 

> There was moderate sample attrition, which differed somewhat between the program and 
control groups: outcome data were obtained for 80% of lottery winners and 72% of 
lottery losers. The study report provides some reassurance that the attrition did not 
undermine the equivalence of the two groups and lead to erroneous results, as follows: (i) 
the study analysis shows that the estimated effects change little even under the 
conservative assumption that the control group’s greater attrition was due to its loss of 
high-achieving students; and (ii) as noted above, lottery winners and losers in the three- 
year follow-up sample (i.e., after sample attrition) were similar in their observable pre- 
program characteristics. Nevertheless, the attrition conceivably could have caused 
unobservable differences between the two groups, possibly leading to inaccurate 
estimates of the program’s effects. 


4 This is known as the “treatment-on-treated” effect, and was calculated using Bloom’s “no-show adjustment.” This 
adjustment relies on the reasonably-safe assumption that winning the lottery had no effect on educational outcomes 
for the 3 1 % of lottery winner who never enrolled in the Promise Academy - i.e., the no-shows. More information 
on the no-show adjustment can be found in Bloom 1984 and Orr 1999, referenced at the end of this summary. 
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> The New York state tests used to measured achievement outcomes have been criticized 
as being susceptible to preparation that improves students’ test-specific skills but not 
necessarily their general knowledge or ability. 5 Ideally, the study would have also 
measured achievement using another test whose validity is well-established. 6 

> This study was conducted at a single site - i.e., one school in New York City. The Top 
Tier initiative’s expert advisory panel believes that replication of the above findings in a 
second randomized controlled trial, conducted in another setting by the same or other 
researchers, would be desirable to (i) rule out the possibility that the findings occurred by 
chance; and (ii) confirm that this intervention is effective in other settings where it woidd 
normally be implemented. 

IV. Summary of the Intervention’s Benefits and Costs: 

If taxpayers fund the delivery of this intervention, what benefits to society can they expect to result, 
and what would be their net cost? The following table provides a summary. This is intended to be a 
general overview of social benefits in relation to taxpayer cost, rather than a comprehensive benefit- 
cost analysis. It assigns monetary value to particular benefits and costs only when doing so requires 
minimal assumptions. The monetary amounts shown are in 2009 dollars. 


Benefits To Society 

■ In 8 th grade (three years after random assignment), Academy students scored IV 2 grade levels 
higher in math and 3 A of a grade level higher in English language arts than control group students. 

Cost To Taxpayers 

■ The Academy cost $19,272 per pupil during the 2008-2009 school year, compared to an average of 
$16,144 per pupil in New York City middle schools during that year. 
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